{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Za8-Nr5k11fh"
      },
      "source": [
        "##### Copyright 2018 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "Eq10uEbw0E4l"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "oYM61xrTsP5d"
      },
      "source": [
        "# Rock, Paper & Scissors with TensorFlow Hub - TFLite"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pAI9358VyAsE"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_lite/tflite_c05_exercise_rock_paper_scissors.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />\n",
        "    Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/examples/blob/master/courses/udacity_intro_to_tensorflow_lite/tflite_c05_exercise_rock_paper_scissors.ipynb\">\n",
        "    <img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />\n",
        "    View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "bL54LWCHt5q5"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "dlauq-4FWGZM"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "import matplotlib.pylab as plt\n",
        "import numpy as np\n",
        "\n",
        "import tensorflow as tf\n",
        "import tensorflow_hub as hub\n",
        "\n",
        "print(\"Version: \", tf.__version__)\n",
        "print(\"Eager mode: \", tf.executing_eagerly())\n",
        "print(\"Hub version: \", hub.__version__)\n",
        "print(\"GPU is\", \"available\" if tf.test.is_gpu_available() else \"NOT AVAILABLE\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "mmaHHH7Pvmth"
      },
      "source": [
        "## Select the Hub/TF2 module to use\n",
        "\n",
        "Hub modules for TF 1.x won't work here, please use one of the selections provided."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "FlsEcKVeuCnf"
      },
      "outputs": [],
      "source": [
        "module_selection = (\"mobilenet_v2\", 224, 1280) #@param [\"(\\\"mobilenet_v2\\\", 224, 1280)\", \"(\\\"inception_v3\\\", 299, 2048)\"] {type:\"raw\", allow-input: true}\n",
        "handle_base, pixels, FV_SIZE = module_selection\n",
        "MODULE_HANDLE =\"https://tfhub.dev/google/tf2-preview/{}/feature_vector/4\".format(handle_base)\n",
        "IMAGE_SIZE = (pixels, pixels)\n",
        "print(\"Using {} with input size {} and output dimension {}\".format(\n",
        "  MODULE_HANDLE, IMAGE_SIZE, FV_SIZE))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "sYUsgwCBv87A"
      },
      "source": [
        "## Data preprocessing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "8nqVX3KYwGPh"
      },
      "source": [
        "Use [TensorFlow Datasets](http://tensorflow.org/datasets) to load the rock, paper and scissors dataset.\n",
        "\n",
        "This `tfds` package is the easiest way to load pre-defined data. If you have your own data, and are interested in importing using it with TensorFlow see [loading image data](../load_data/images.ipynb)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "jGvpkDj4wBup"
      },
      "outputs": [],
      "source": [
        "import tensorflow_datasets as tfds\n",
        "tfds.disable_progress_bar()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "YkF4Boe5wN7N"
      },
      "source": [
        "The `tfds.load` method downloads and caches the data, and returns a `tf.data.Dataset` object. These objects provide powerful, efficient methods for manipulating data and piping it into your model.\n",
        "\n",
        "Since `\"rock_paper_scissors\"` doesn't define standard splits, use the subsplit feature to divide it into (train, validation, test) with 80%, 10%, 10% of the data respectively."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "SQ9xK9F2wGD8"
      },
      "outputs": [],
      "source": [
        "splits = tfds.Split.ALL.subsplit(weighted=(80, 10, 10))\n",
        "\n",
        "# Go to the TensorFlow Dataset's website and search for the Rock, Paper, Scissors dataset and load it here\n",
        "splits, info = tfds.load( # YOUR CODE HERE )\n",
        "    \n",
        "# Save the dataset splits in a tuple\n",
        "(train_examples, validation_examples, test_examples) = splits\n",
        "\n",
        "num_examples = info.splits['train'].num_examples\n",
        "num_classes = info.features['label'].num_classes"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "pmXQYXNWwf19"
      },
      "source": [
        "### Format the Data\n",
        "\n",
        "Use the `tf.image` module to format the images for the task.\n",
        "\n",
        "Resize the images to a fixes input size, and rescale the input channels"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "y7UyXblSwkUS"
      },
      "outputs": [],
      "source": [
        "def format_image(image, label):\n",
        "  image = tf.image.resize(image, IMAGE_SIZE) / 255.0\n",
        "  return  image, label\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "1nrDR8CnwrVk"
      },
      "source": [
        "Now shuffle and batch the data\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "zAEUG7vawxLm"
      },
      "outputs": [],
      "source": [
        "BATCH_SIZE = 32 #@param {type:\"integer\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "fHEC9mbswxvM"
      },
      "outputs": [],
      "source": [
        "# Prepare the examples by preprocessing them and then batching them (and optionally prefetching them)\n",
        "\n",
        "# If you wish you can shuffle train set here\n",
        "train_batches = # YOUR CODE HERE\n",
        "\n",
        "validation_batches = # YOUR CODE HERE\n",
        "test_batches = # YOUR CODE HERE"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "ghQhZjgEw1cK"
      },
      "source": [
        "Inspect a batch"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "gz0xsMCjwx54"
      },
      "outputs": [],
      "source": [
        "for image_batch, label_batch in train_batches.take(1):\n",
        "  pass\n",
        "\n",
        "image_batch.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "FS_gVStowW3G"
      },
      "source": [
        "## Defining the model\n",
        "\n",
        "All it takes is to put a linear classifier on top of the `feature_extractor_layer` with the Hub module.\n",
        "\n",
        "For speed, we start out with a non-trainable `feature_extractor_layer`, but you can also enable fine-tuning for greater accuracy."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "RaJW3XrPyFiF"
      },
      "outputs": [],
      "source": [
        "do_fine_tuning = False #@param {type:\"boolean\"}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "50FYNIb1dmJH"
      },
      "outputs": [],
      "source": [
        "# Build the model with a TFHub KerasLayer and attach a classification head to it\n",
        "print(\"Building model with\", MODULE_HANDLE)\n",
        "model = tf.keras.Sequential([\n",
        "    hub.KerasLayer(MODULE_HANDLE,\n",
        "                   input_shape=IMAGE_SIZE + (3, ), \n",
        "                   output_shape=[FV_SIZE],\n",
        "                   trainable=do_fine_tuning),\n",
        "    tf.keras.layers.Dense(num_classes)\n",
        "])\n",
        "model.summary()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "u2e5WupIw2N2"
      },
      "source": [
        "## Training the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "9f3yBUvkd_VJ"
      },
      "outputs": [],
      "source": [
        "if do_fine_tuning:\n",
        "  model.compile(\n",
        "    optimizer=tf.keras.optimizers.SGD(lr=0.002, momentum=0.9), \n",
        "    loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "    metrics=['accuracy'])\n",
        "else:\n",
        "  model.compile(\n",
        "    optimizer='adam', \n",
        "    loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        "    metrics=['accuracy'])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "w_YKX2Qnfg6x"
      },
      "outputs": [],
      "source": [
        "EPOCHS = 3\n",
        "hist = model.fit(train_batches,\n",
        "                    epochs=EPOCHS,\n",
        "                    validation_data=validation_batches)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "u_psFoTeLpHU"
      },
      "source": [
        "## Export the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "XaSb5nVzHcVv"
      },
      "outputs": [],
      "source": [
        "RPS_SAVED_MODEL = \"exp_saved_model\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "fZqRAg1uz1Nu"
      },
      "source": [
        "Export the SavedModel"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "yJMue5YgnwtN"
      },
      "outputs": [],
      "source": [
        "# Use TensorFlow's SavedModel API to export the SavedModel from the trained Keras model\n",
        "# YOUR CODE HERE"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "uVrnAWjqCMxs"
      },
      "source": [
        "Here you can verify the default signature of your exported SavedModel"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "SOQF4cOan0SY"
      },
      "outputs": [],
      "source": [
        "%%bash -s $RPS_SAVED_MODEL\n",
        "saved_model_cli show --dir $1 --tag_set serve --signature_def serving_default"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "FY7QGBgBytwX"
      },
      "outputs": [],
      "source": [
        "loaded = tf.saved_model.load(RPS_SAVED_MODEL)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "tIhPyMISz952"
      },
      "outputs": [],
      "source": [
        "print(list(loaded.signatures.keys()))\n",
        "infer = loaded.signatures[\"serving_default\"]\n",
        "print(infer.structured_input_signature)\n",
        "print(infer.structured_outputs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "XxLiLC8n0H16"
      },
      "source": [
        "## Convert with TFLiteConverter"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "WmSr2-yZoUhz"
      },
      "outputs": [],
      "source": [
        "# Intialize the TFLite converter to load the SavedModel\n",
        "converter = # YOUR CODE HERE\n",
        "# Set the optimization strategy for 'size' in the converter \n",
        "converter.optimizations = [# YOUR CODE HERE]\n",
        "\n",
        "# Use the tool to finally convert the model\n",
        "tflite_model = # YOUR CODE HERE\n",
        "    \n",
        "with open(\"converted_model.tflite\", \"wb\") as f:\n",
        "  f.write(tflite_model)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "BbTF6nd1KG2o"
      },
      "source": [
        "Run the following cells to check whether your TFLite model is working using the Python Interpreter"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "dg2NkVTmLUdJ"
      },
      "outputs": [],
      "source": [
        "#@title Loading the converted TFLite model...\n",
        "# Load TFLite model and allocate tensors.\n",
        "tflite_model_file = 'converted_model.tflite'\n",
        "with open(tflite_model_file, 'rb') as fid:\n",
        "  tflite_model = fid.read()\n",
        "  \n",
        "interpreter = tf.lite.Interpreter(model_content=tflite_model)\n",
        "interpreter.allocate_tensors()\n",
        "\n",
        "input_index = interpreter.get_input_details()[0][\"index\"]\n",
        "output_index = interpreter.get_output_details()[0][\"index\"]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "snJQVs9JNglv"
      },
      "outputs": [],
      "source": [
        "#@title Testing on random test examples...\n",
        "from tqdm import tqdm\n",
        "\n",
        "# Gather results for the randomly sampled test images\n",
        "predictions = []\n",
        "\n",
        "test_labels, test_imgs = [], []\n",
        "for img, label in tqdm(test_batches.take(10)):\n",
        "  interpreter.set_tensor(input_index, img)\n",
        "  interpreter.invoke()\n",
        "  predictions.append(interpreter.get_tensor(output_index))\n",
        "  \n",
        "  test_labels.append(label.numpy()[0])\n",
        "  test_imgs.append(img)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "YMTWNqPpNiAI"
      },
      "outputs": [],
      "source": [
        "#@title Utility functions for plotting\n",
        "# Utilities for plotting\n",
        "\n",
        "class_names = ['rock', 'paper', 'scissors']\n",
        "\n",
        "def plot_image(i, predictions_array, true_label, img):\n",
        "  predictions_array, true_label, img = predictions_array[i], true_label[i], img[i]\n",
        "  plt.grid(False)\n",
        "  plt.xticks([])\n",
        "  plt.yticks([])\n",
        "    \n",
        "  img = np.squeeze(img)\n",
        "\n",
        "  plt.imshow(img, cmap=plt.cm.binary)\n",
        "\n",
        "  predicted_label = np.argmax(predictions_array)\n",
        "  if predicted_label == true_label:\n",
        "    color = 'green'\n",
        "  else:\n",
        "    color = 'red'\n",
        "  \n",
        "  plt.xlabel(\"{} {:2.0f}% ({})\".format(class_names[predicted_label],\n",
        "                                100*np.max(predictions_array),\n",
        "                                class_names[true_label]),\n",
        "                                color=color)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "cellView": "form",
        "colab": {},
        "colab_type": "code",
        "id": "1-lbnicPNkZs"
      },
      "outputs": [],
      "source": [
        "#@title Visualize the outputs { run: \"auto\" }\n",
        "index = 9 #@param {type:\"slider\", min:0, max:9, step:1}\n",
        "plt.figure(figsize=(6,3))\n",
        "plt.subplot(1,2,1)\n",
        "plot_image(index, predictions, test_labels, test_imgs)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "PmZRieHmKLY5"
      },
      "source": [
        "Download the model\n",
        "\n",
        "**NOTE: You might have to run to the cell below twice**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "0jJAxrQB2VFw"
      },
      "outputs": [],
      "source": [
        "with open('labels.txt', 'w') as f:\n",
        "  f.write('\\n'.join(class_names))\n",
        "\n",
        "\n",
        "try:\n",
        "  from google.colab import files\n",
        "  files.download('converted_model.tflite')  \n",
        "  files.download('labels.txt')\n",
        "except:\n",
        "  pass"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "BDlmpjC6VnFZ"
      },
      "source": [
        "# Prepare the test images for download (Optional)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "_1ja_WA0WZOH"
      },
      "source": [
        "This part involves downloading additional test images for the Mobile Apps only in case you need to try out more samples"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "fzLKEBrfTREA"
      },
      "outputs": [],
      "source": [
        "!mkdir -p test_images"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Qn7ukNQCSewb"
      },
      "outputs": [],
      "source": [
        "from PIL import Image\n",
        "\n",
        "for index, (image, label) in enumerate(test_batches.take(50)):\n",
        "  image = tf.cast(image * 255.0, tf.uint8)\n",
        "  image = tf.squeeze(image).numpy()\n",
        "  pil_image = Image.fromarray(image)\n",
        "  pil_image.save('test_images/{}_{}.jpg'.format(class_names[label[0]], index))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "xVKKWUG8UMO5"
      },
      "outputs": [],
      "source": [
        "!ls test_images"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "l_w_-UdlS9Vi"
      },
      "outputs": [],
      "source": [
        "!zip -qq rps_test_images.zip -r test_images/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 0,
      "metadata": {
        "colab": {},
        "colab_type": "code",
        "id": "Giva6EHwWm6Y"
      },
      "outputs": [],
      "source": [
        "try:\n",
        "  files.download('rps_test_images.zip')\n",
        "except:\n",
        "  pass"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "tflite_c05_exercise_rock_paper_scissors.ipynb",
      "private_outputs": true,
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
